Any examples of loading to BigQuery using a POST request and Java client library? -
does have examples of creating new insert job bigquery using both:
- the bigquery java client library
- creating load job post request documented here: https://developers.google.com/bigquery/loading-data-into-bigquery#loaddatapostrequest
you need call bigquery.jobs().insert(...) method.
i don't know have done yet should have authenticated client api @ least like:
bigquery = new bigquery.builder(http_transport, json_factory, credentials) .setapplicationname("...").build();
that's simplified version of insertrows method wrote using google-http-client library java , bigquery-api (you should check dataset exists, validate ids etc.):
public long insertrows(string projectid, string datasetid, string tableid, inputstream schema, abstractinputstreamcontent data) { try { // defining table fields objectmapper mapper = new objectmapper(); list<tablefieldschema> schemafields = mapper.readvalue(schema, new typereference<list<tablefieldschema>>(){}); tableschema tableschema = new tableschema().setfields(schemafields); // table reference tablereference tablereference = new tablereference() .setprojectid(projectid) .setdatasetid(datasetid) .settableid(tableid); // load job configuration jobconfigurationload loadconfig = new jobconfigurationload() .setdestinationtable(tablereference) .setschema(tableschema) // data in json format (could csv) .setsourceformat("newline_delimited_json") // table created if not exists .setcreatedisposition("create_if_needed") // append data (not override data) .setwritedisposition("write_append"); // if data coming google cloud storage //.setsourceuris(...); // load job job loadjob = new job() .setjobreference( new jobreference() .setjobid(joiner.on("-").join("insert", projectid, datasetid, tableid, datetime.now().tostring("dd-mm-yyyy_hh-mm-ss-sss"))) .setprojectid(projectid)) .setconfiguration(new jobconfiguration().setload(loadconfig)); // job execution job createtablejob = bigquery.jobs().insert(projectid, loadjob, data).execute(); // if loading data google cloud storage //createtablejob = bigquery.jobs().insert(projectid, loadjob).execute(); string jobid = createtablejob.getjobreference().getjobid(); // wait job completion createtablejob = waitforjob(projectid, createtablejob); long rowcount = createtablejob != null ? createtablejob.getstatistics().getload().getoutputrows() : 0l; log.info("{} rows inserted in table '{}' (dataset: '{}', project: '{}')", rowcount, tableid, datasetid, projectid); return rowcount; } catch (ioexception e) { throw throwables.propagate(e); } }
i don't know format of data if using files, can add function like:
public long insertrows(string projectid, string datasetid, string tableid, file schema, file data) { try { return insertrows(projectid, datasetid, tableid, new fileinputstream(schema), new filecontent(mediatype.octet_stream.tostring(), data)); } catch (filenotfoundexception e) { throw throwables.propagate(e); } }
Comments
Post a Comment