fiware - HTTPBadRequestException: orion/0.19.0 libcurl/7.19.7 user agent not supported and Cosmos HttpFS response: HTTP/1.1 401 Unauthorized -

good afternoon,

i'm trying configure persistence of orion context's in cosmos hdfs. far have created orion context broker instance (orion-psb-image-r4.2), created cosmos user account , folder 'cardio' inside (/user/username/cardio).

i've configured cygnus.conf file (/usr/local/apache-flume-1.4.0-bin/conf/cygnus.conf) following:

# apache_flume_home/conf/cygnus.conf orionagent.sources = http-source orionagent.sinks = hdfs-sink orionagent.channels = notifications  # flume source, must not changed orionagent.sources.http-source.type = org.apache.flume.source.http.httpsource # channel name write notification events orionagent.sources.http-source.channels = notifications # listening port flume source use receiving incoming notifications orionagent.sources.http-source.port = 5050 # flume handler parse notifications, must not changed orionagent.sources.http-source.handler = es.tid.fiware.orionconnectors.cosmosinjector.orionresthandler # regular expression orion version notifications have in headers orionagent.sources.http-source.handler.orion_version = 0\.11\.* # url target orionagent.sources.http-source.handler.notification_target = /notify  # channel name read notification events orionagent.sinks.hdfs-sink.channel = notifications # flume sink process , persist in hdfs notification events, must not changed orionagent.sinks.hdfs-sink.type = es.tid.fiware.orionconnectors.cosmosinjector.orionhdfssink # ip address of cosmos deployment notification events persisted orionagent.sinks.hdfs-sink.cosmos_host = 130.206.80.46 # port of cosmos service listening persistence operations; 14000 httpfs, 50070 webhdfs , free choice inifinty orionagent.sinks.hdfs-sink.cosmos_port = 14000 # username allowed write in hdfs (/user/myusername) orionagent.sinks.hdfs-sink.cosmos_username = username # dataset persist data (/user/myusername/mydataset) orionagent.sinks.hdfs-sink.cosmos_dataset = cardio # hdfs backend type (webhdfs, httpfs or infinity) orionagent.sinks.hdfs-sink.hdfs_api = httpfs  # channel name orionagent.channels.notifications.type = memory # capacity of channel orionagent.channels.notifications.capacity = 1000 # amount of bytes can sent per transaction orionagent.channels.notifications.transactioncapacity = 100 

the issue when start agent following command:

/usr/local/apache-flume-1.4.0-bin/bin/flume-ng agent --conf /usr/local/apache-flume-1.4.0-bin/conf/ -f /usr/local/apache-flume-1.4.0-bin/conf/cygnus.conf -n orionagent -dflume.root.logger=info,consoleapache-flume-1.4.0-bin 

i following error messages:

2015-07-17 15:33:10,477 (lifecyclesupervisor-1-1) [info - es.tid.fiware.orionconnectors.cosmosinjector.hdfs.httpfsbackend.createdir(httpfsbackend.java:68)] httpfs operation: put http://130.206.80.46:14000/webhdfs/v1/user/arlourenco/cardio/?op=mkdirs&user.name=arlourenco http/1.1 2015-07-17 15:33:10,708 (lifecyclesupervisor-1-1) [info - es.tid.fiware.orionconnectors.cosmosinjector.hdfs.httpfsbackend.createdir(httpfsbackend.java:71)] httpfs response: http/1.1 401 unauthorized 2015-07-17 15:33:10,708 (lifecyclesupervisor-1-1) [error - es.tid.fiware.orionconnectors.cosmosinjector.orionhdfssink.start(orionhdfssink.java:108)]  directory not created in hdfs. httpfs response: 401 unauthorized 2015-07-17 15:33:17,470 (910628961@qtp-586740701-0) [warn - org.apache.flume.source.http.httpsource$flumehttpservlet.dopost(httpsource.java:186)] received bad request client. 

and:

org.apache.flume.source.http.httpbadrequestexception: orion/0.19.0 libcurl/7.19.7 user agent not supported 

here complete log:

+ exec /usr/lib/jvm/java-1.6.0-openjdk.x86_64/bin/java -xmx20m -dflume.root.logger=info,console -cp '/usr/local/apache-flume-1.4.0-bin/conf:/usr/local/apache-flume-1.4.0-bin/lib/*' -djava.library.path= org.apache.flume.node.application -f /usr/local/apache-flume-1.4.0-bin/conf/cygnus.conf -n orionagent 2015-07-17 15:33:09,858 (lifecyclesupervisor-1-0) [info - org.apache.flume.node.pollingpropertiesfileconfigurationprovider.start(pollingpropertiesfileconfigurationprovider.java:61)] configuration provider starting 2015-07-17 15:33:09,863 (conf-file-poller-0) [info - org.apache.flume.node.pollingpropertiesfileconfigurationprovider$filewatcherrunnable.run(pollingpropertiesfileconfigurationprovider.java:133)] reloading configuration file:/usr/local/apache-flume-1.4.0-bin/conf/cygnus.conf 2015-07-17 15:33:09,871 (conf-file-poller-0) [info - org.apache.flume.conf.flumeconfiguration$agentconfiguration.addproperty(flumeconfiguration.java:930)] added sinks: hdfs-sink agent: orionagent 2015-07-17 15:33:09,871 (conf-file-poller-0) [info - org.apache.flume.conf.flumeconfiguration$agentconfiguration.addproperty(flumeconfiguration.java:1016)] processing:hdfs-sink 2015-07-17 15:33:09,872 (conf-file-poller-0) [info - org.apache.flume.conf.flumeconfiguration$agentconfiguration.addproperty(flumeconfiguration.java:1016)] processing:hdfs-sink 2015-07-17 15:33:09,872 (conf-file-poller-0) [info - org.apache.flume.conf.flumeconfiguration$agentconfiguration.addproperty(flumeconfiguration.java:1016)] processing:hdfs-sink 2015-07-17 15:33:09,872 (conf-file-poller-0) [info - org.apache.flume.conf.flumeconfiguration$agentconfiguration.addproperty(flumeconfiguration.java:1016)] processing:hdfs-sink 2015-07-17 15:33:09,872 (conf-file-poller-0) [info - org.apache.flume.conf.flumeconfiguration$agentconfiguration.addproperty(flumeconfiguration.java:1016)] processing:hdfs-sink 2015-07-17 15:33:09,872 (conf-file-poller-0) [info - org.apache.flume.conf.flumeconfiguration$agentconfiguration.addproperty(flumeconfiguration.java:1016)] processing:hdfs-sink 2015-07-17 15:33:09,872 (conf-file-poller-0) [info - org.apache.flume.conf.flumeconfiguration$agentconfiguration.addproperty(flumeconfiguration.java:1016)] processing:hdfs-sink 2015-07-17 15:33:09,873 (conf-file-poller-0) [info - org.apache.flume.conf.flumeconfiguration$agentconfiguration.addproperty(flumeconfiguration.java:1016)] processing:hdfs-sink 2015-07-17 15:33:09,885 (conf-file-poller-0) [info - org.apache.flume.conf.flumeconfiguration.validateconfiguration(flumeconfiguration.java:140)] post-validation flume configuration contains configuration agents: [orionagent] 2015-07-17 15:33:09,885 (conf-file-poller-0) [info - org.apache.flume.node.abstractconfigurationprovider.loadchannels(abstractconfigurationprovider.java:150)] creating channels 2015-07-17 15:33:09,899 (conf-file-poller-0) [info - org.apache.flume.channel.defaultchannelfactory.create(defaultchannelfactory.java:40)] creating instance of channel notifications type memory 2015-07-17 15:33:09,908 (conf-file-poller-0) [info - org.apache.flume.node.abstractconfigurationprovider.loadchannels(abstractconfigurationprovider.java:205)] created channel notifications 2015-07-17 15:33:09,912 (conf-file-poller-0) [info - org.apache.flume.source.defaultsourcefactory.create(defaultsourcefactory.java:39)] creating instance of source http-source, type org.apache.flume.source.http.httpsource 2015-07-17 15:33:09,961 (conf-file-poller-0) [info - org.apache.flume.sink.defaultsinkfactory.create(defaultsinkfactory.java:40)] creating instance of sink: hdfs-sink, type: es.tid.fiware.orionconnectors.cosmosinjector.orionhdfssink 2015-07-17 15:33:09,967 (conf-file-poller-0) [info - org.apache.flume.node.abstractconfigurationprovider.getconfiguration(abstractconfigurationprovider.java:119)] channel notifications connected [http-source, hdfs-sink] 2015-07-17 15:33:09,972 (conf-file-poller-0) [info - org.apache.flume.node.application.startallcomponents(application.java:138)] starting new configuration:{ sourcerunners:{http-source=eventdrivensourcerunner: { source:org.apache.flume.source.http.httpsource{name:http-source,state:idle} }} sinkrunners:{hdfs-sink=sinkrunner: { policy:org.apache.flume.sink.defaultsinkprocessor@e8234ec countergroup:{ name:null counters:{} } }} channels:{notifications=org.apache.flume.channel.memorychannel{name: notifications}} } 2015-07-17 15:33:09,973 (conf-file-poller-0) [info - org.apache.flume.node.application.startallcomponents(application.java:145)] starting channel notifications 2015-07-17 15:33:10,027 (lifecyclesupervisor-1-0) [info - org.apache.flume.instrumentation.monitoredcountergroup.register(monitoredcountergroup.java:110)] monitoried counter group type: channel, name: notifications, registered successfully. 2015-07-17 15:33:10,028 (lifecyclesupervisor-1-0) [info - org.apache.flume.instrumentation.monitoredcountergroup.start(monitoredcountergroup.java:94)] component type: channel, name: notifications started 2015-07-17 15:33:10,028 (conf-file-poller-0) [info - org.apache.flume.node.application.startallcomponents(application.java:173)] starting sink hdfs-sink 2015-07-17 15:33:10,029 (conf-file-poller-0) [info - org.apache.flume.node.application.startallcomponents(application.java:184)] starting source http-source 2015-07-17 15:33:10,069 (lifecyclesupervisor-1-3) [info - org.mortbay.log.slf4jlog.info(slf4jlog.java:67)] logging org.slf4j.impl.log4jloggeradapter(org.mortbay.log) via org.mortbay.log.slf4jlog 2015-07-17 15:33:10,149 (lifecyclesupervisor-1-3) [info - org.mortbay.log.slf4jlog.info(slf4jlog.java:67)] jetty-6.1.26 2015-07-17 15:33:10,306 (lifecyclesupervisor-1-3) [info - org.mortbay.log.slf4jlog.info(slf4jlog.java:67)] started socketconnector@0.0.0.0:5050 2015-07-17 15:33:10,307 (lifecyclesupervisor-1-3) [info - org.apache.flume.instrumentation.monitoredcountergroup.register(monitoredcountergroup.java:110)] monitoried counter group type: source, name: http-source, registered successfully. 2015-07-17 15:33:10,307 (lifecyclesupervisor-1-3) [info - org.apache.flume.instrumentation.monitoredcountergroup.start(monitoredcountergroup.java:94)] component type: source, name: http-source started 2015-07-17 15:33:10,456 (lifecyclesupervisor-1-1) [info - es.tid.fiware.orionconnectors.cosmosinjector.http.httpclientfactory.<init>(httpclientfactory.java:68)] setting max total connections 500 , default max connections per route 100 2015-07-17 15:33:10,477 (lifecyclesupervisor-1-1) [info - es.tid.fiware.orionconnectors.cosmosinjector.hdfs.httpfsbackend.createdir(httpfsbackend.java:68)] httpfs operation: put http://130.206.80.46:14000/webhdfs/v1/user/arlourenco/cardio/?op=mkdirs&user.name=arlourenco http/1.1 2015-07-17 15:33:10,708 (lifecyclesupervisor-1-1) [info - es.tid.fiware.orionconnectors.cosmosinjector.hdfs.httpfsbackend.createdir(httpfsbackend.java:71)] httpfs response: http/1.1 401 unauthorized 2015-07-17 15:33:10,708 (lifecyclesupervisor-1-1) [error - es.tid.fiware.orionconnectors.cosmosinjector.orionhdfssink.start(orionhdfssink.java:108)]  directory not created in hdfs. httpfs response: 401 unauthorized 2015-07-17 15:33:17,470 (910628961@qtp-586740701-0) [warn - org.apache.flume.source.http.httpsource$flumehttpservlet.dopost(httpsource.java:186)] received bad request client. org.apache.flume.source.http.httpbadrequestexception: orion/0.19.0 libcurl/7.19.7 user agent not supported         @ es.tid.fiware.orionconnectors.cosmosinjector.orionresthandler.getevents(orionresthandler.java:93)         @ org.apache.flume.source.http.httpsource$flumehttpservlet.dopost(httpsource.java:184)         @ javax.servlet.http.httpservlet.service(httpservlet.java:725)         @ javax.servlet.http.httpservlet.service(httpservlet.java:814)         @ org.mortbay.jetty.servlet.servletholder.handle(servletholder.java:511)         @ org.mortbay.jetty.servlet.servlethandler.handle(servlethandler.java:401)         @ org.mortbay.jetty.servlet.sessionhandler.handle(sessionhandler.java:182)         @ org.mortbay.jetty.handler.contexthandler.handle(contexthandler.java:766)         @ org.mortbay.jetty.handler.handlerwrapper.handle(handlerwrapper.java:152)         @ org.mortbay.jetty.server.handle(server.java:326)         @ org.mortbay.jetty.httpconnection.handlerequest(httpconnection.java:542)         @ org.mortbay.jetty.httpconnection$requesthandler.content(httpconnection.java:945)         @ org.mortbay.jetty.httpparser.parsenext(httpparser.java:756)         @ org.mortbay.jetty.httpparser.parseavailable(httpparser.java:218)         @ org.mortbay.jetty.httpconnection.handle(httpconnection.java:404)         @ org.mortbay.jetty.bio.socketconnector$connection.run(socketconnector.java:228)         @ org.mortbay.thread.queuedthreadpool$poolthread.run(queuedthreadpool.java:582) 

what issue?

first of all, using pretty old version of cygnus; guess 1 pre-installed in orion-psb-image-r4.2, 0.5.2. latest 1 0.8.2, please consider upgrade, specially after enabling oauth2 authentication/authorization in cosmos' webhdfs (you can find detailed instructions upgrading here , detailed instructions using oauth2 webhdfs here).

anyway, problem sending user agent header different orion/*, i.e. libcurl/7.19.7. possible avoid sending such user agent client? if not, cygnus code should modified in order accept kind of user agents (something pointed other users before, admit useful).