httpresponse - Receive partial file(sometimes) when reading from Google Storage using HTTP Response -
i trying read files google storage , write files in our filesystem (hdfs). if run period of time (lets 7 days), full file lines matching whats on source , partial files (discrepancy quite large). pasting below method takes response , writes file. or suggestions how can troubleshoot further appreciated.
thanks,
before calling method simple check on response status code -
if(response.getstatuscode() == 200 && stringutils.equals(response.getcontenttype(), "application/zip")) { writehdfsfile(response, path); } private void writehdfsfile(httpresponse response, string path) throws ioexception { final gzipinputstream inputstream = new gzipinputstream(response.getcontent()); path filepath = new path(path); final fsdataoutputstream outputstream = filesystem.create(filepath, true); final byte[] buffer = new byte[1024]; int length; try { while((length = inputstream.read(buffer)) > 0) { outputstream.write(buffer, 0, length); } outputstream.flush(); } { inputstream.close(); outputstream.close(); } }
the way solved downloading file first , unzipping , writing it. basically, splitting 2 steps solved issue. if else ran same issue..
Comments
Post a Comment