since last update , haven't been able upload data cosmos using cygnus . aware need use oauth2 token . did request token .
curl -k -x post "https://cosmos.lab.fiware.org:13000/cosmos-auth/v1/token" -h "content-type: application/x-www-form-urlencoded" -d "grant_type=password&username=guillaume.jourdain@4planet.eu&password=xxxxx" i token, try check token :
curl -x "http://cosmos.lab.fiware.org:14000/webhdfs/v1/guillaume.jourdain/hostabee?op=liststatus&user.name=guillaume.jourdain@4planet.eu" -h "x-auth-token: thetoken" and :
curl -x "http://cosmos.lab.fiware.org:14000/webhdfs/v1/guillaume.jourdain/hostabee?op=liststatus&user.name=guillaume.jourdain" -h "x-auth-token: thetoken" and everytime , each of command , token tried :
user token not authorized
next tried put oauth parameter in cygnus conf file , occured everytime :
2015-07-17 16:17:17,797 (lifecyclesupervisor-1-1) [info - es.tid.fiware.orionconnectors.cosmosinjector.hdfs.httpfsbackend.createdir(httpfsbackend.java:71)] httpfs response: http/1.1 401 unauthorized 2015-07-17 16:17:17,798 (lifecyclesupervisor-1-1) [error - es.tid.fiware.orionconnectors.cosmosinjector.orionhdfssink.start(orionhdfssink.java:108)] directory not created in hdfs. httpfs response: 401 unauthorized so yeah , moment i'm kinda stuck . have information me resolve problem ?
edit :
here's cygnus configuration file , maybe problem located here
apache_flume_home/conf/cygnus.conf orionagent.sources = http-source orionagent.sinks = hdfs-sink orionagent.channels = notifications # flume source, must not changed orionagent.sources.http-source.type = org.apache.flume.source.http.httpsource # channel name write notification events orionagent.sources.http-source.channels = notifications # listening port flume source use receiving incoming notifications orionagent.sources.http-source.port = 5050 # flume handler parse notifications, must not changed orionagent.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.orionresthandler # regular expression orion version notifications have in headers orionagent.sources.http-source.handler.orion_version = 0\.23\.* # url target orionagent.sources.http-source.handler.notification_target = /notify # channel name read notification events orionagent.sinks.hdfs-sink.channel = notifications # flume sink process , persist in hdfs notification events, must not changed orionagent.sinks.hdfs-sink.type = com.telefonica.iot.cygnus.sinks.orionhdfssink # ip address of cosmos deployment notification events persisted orionagent.sinks.hdfs-sink.cosmos_host = 130.206.80.46 # port of cosmos service listening persistence operations; 14000 httpfs, 50070 webhdfs , free choice inifinty orionagent.sinks.hdfs-sink.cosmos_port = 14000 # username allowed write in hdfs (/user/myusername) orionagent.sinks.hdfs-sink.cosmos_username = guillaume.jourdain # dataset persist data (/user/myusername/mydataset) orionagent.sinks.hdfs-sink.cosmos_password = xxxxx orionagent.sinks.hdfs-sink.cosmos_dataset = hostabee orionagent.sinks.hdfs-sink.attr_persistence = column orionagent.sinks.hdfs-sink.hive_host = 130.206.80.46 orionagent.sinks.hdfs-sink.hive_port = 10000 orionagent.sinks.hdfs-sink.oauth2_token = thetoken # hdfs backend type (webhdfs, httpfs or infinity) orionagent.sinks.hdfs-sink.hdfs_api = webhdfs # channel name orionagent.channels.notifications.type = memory # capacity of channel orionagent.channels.notifications.capacity = 1000 # amount of bytes can sent per transaction orionagent.channels.notifications.transactioncapacity = 100 now error (and others). sink , handlers does'nt seems found
2015-07-27 14:27:10,562 (conf-file-poller-0) [info - org.apache.flume.sink.defaultsinkfactory.create(defaultsinkfactory.java:40)] creating instance of sink: hdfs-sink, type: com.telefonica.iot.cygnus.sinks.orionhdfssink 2015-07-27 14:27:10,562 (conf-file-poller-0) [error - org.apache.flume.node.pollingpropertiesfileconfigurationprovider$filewatcherrunnable.run(pollingpropertiesfileconfigurationprovider.java:142)] failed load configuration data. exception follows. org.apache.flume.flumeexception: unable load sink type: com.telefonica.iot.cygnus.sinks.orionhdfssink, class: com.telefonica.iot.cygnus.sinks.orionhdfssink @ org.apache.flume.sink.defaultsinkfactory.getclass(defaultsinkfactory.java:69) @ org.apache.flume.sink.defaultsinkfactory.create(defaultsinkfactory.java:41) @ org.apache.flume.node.abstractconfigurationprovider.loadsinks(abstractconfigurationprovider.java:415) @ org.apache.flume.node.abstractconfigurationprovider.getconfiguration(abstractconfigurationprovider.java:103) @ org.apache.flume.node.pollingpropertiesfileconfigurationprovider$filewatcherrunnable.run(pollingpropertiesfileconfigurationprovider.java:140) @ java.util.concurrent.executors$runnableadapter.call(executors.java:471) @ java.util.concurrent.futuretask$sync.innerrunandreset(futuretask.java:351) @ java.util.concurrent.futuretask.runandreset(futuretask.java:178) @ java.util.concurrent.scheduledthreadpoolexecutor$scheduledfuturetask.access$201(scheduledthreadpoolexecutor.java:165) @ java.util.concurrent.scheduledthreadpoolexecutor$scheduledfuturetask.run(scheduledthreadpoolexecutor.java:267) @ java.util.concurrent.threadpoolexecutor.runworker(threadpoolexecutor.java:1146) @ java.util.concurrent.threadpoolexecutor$worker.run(threadpoolexecutor.java:615) @ java.lang.thread.run(thread.java:701) caused by: java.lang.classnotfoundexception: com.telefonica.iot.cygnus.sinks.orionhdfssink @ java.net.urlclassloader$1.run(urlclassloader.java:217) @ java.security.accesscontroller.doprivileged(native method) @ java.net.urlclassloader.findclass(urlclassloader.java:205) @ java.lang.classloader.loadclass(classloader.java:323) @ sun.misc.launcher$appclassloader.loadclass(launcher.java:294) @ java.lang.classloader.loadclass(classloader.java:268) @ java.lang.class.forname0(native method) @ java.lang.class.forname(class.java:190) @ org.apache.flume.sink.defaultsinkfactory.getclass(defaultsinkfactory.java:67) ... 12 more thank reading .
regarding webhdfs command listing hdfs folder:
curl -x "http://cosmos.lab.fiware.org:14000/webhdfs/v1/guillaume.jourdain/hostabee?op=liststatus&user.name=guillaume.jourdain@4planet.eu" -h "x-auth-token: thetoken" the user.name should user.name=guillaume.jourdain (without @4planet.eu part).
regarding cygnus, have upgraded 0.8.2? the cygnus version supporting oauth2. guess did not upgrade because of es.tid.fiware.orionconnectors.cosmosinjector.orionhdfssink logs (those packages previous 0.8.0). have details upgrading here.
Comments
Post a Comment