4

Right now I'm trying to create a dataset for my orion data with all the entitys info. I set up all correctly and when some data changes orion send a notification to cygnus and then, cygnus will add the data to ckan. As seen in this image.

My arquitecture

The problem is that cygnus create a ckan resource for each entity. What could I do to make cygnus put all data in only one resource? Thanks.

EDIT 1:

I just found that I can simplify by making just one subscription in orion.

{
    "entities": [
        {
            "type": "Room",
            "isPattern": "true",
            "id": "Room.*"
        }
    ] ...

Then I noticed the grouping feature (http://fiware-cygnus.readthedocs.io/en/develop/installation_and_administration_guide/grouping_rules_conf/index.html) in cygnus but right now I'm lost because cygnus don't want to load my grouping config file. It shows this error:

time=2016-05-03T05:32:29.658CDT | lvl=INFO | trans= | srv= | subsrv= | function=<init> | comp=Cygnus | msg=com.telefonica.iot.cygnus.interceptors.GroupingRules[58] : No grouping rules have been read
Exception in thread "Thread-1" java.lang.NullPointerException
    at java.io.File.<init>(File.java:277)
    at com.telefonica.iot.cygnus.interceptors.GroupingInterceptor$ConfigurationReader.run(GroupingInterceptor.java:244)

In my agent config file I have:

cygnusagent.sources.http-source.interceptors.gi.gropuing_rules_conf_file = /usr/cygnus/conf/grouping_rules.conf

And in grouping_rules.conf:

{
    "grouping_rules": [
        {
            "id": 1,
            "fields": [
                "entityId"
            ],
            "regex": "room.*",
            "destination": "Rooms",
            "fiware_service_path": "/myhouse"
        }
    ]
}

Edit 2:

I have all the lines as documentation said:

    cygnusagent.sources = http-source
    cygnusagent.sinks = ckan-sink
    cygnusagent.channels = ckan-channel

    cygnusagent.sources.http-source.channels = ckan-channel
    cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
    cygnusagent.sources.http-source.port = 5050
    cygnusagent.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.OrionRestHandler
    cygnusagent.sources.http-source.handler.notification_target = /notify
    cygnusagent.sources.http-source.handler.default_service = test
    cygnusagent.sources.http-source.handler.default_service_path = /myhouse
    cygnusagent.sources.http-source.handler.events_ttl = 5
    cygnusagent.sources.http-source.interceptors = ts gi
    cygnusagent.sources.http-source.interceptors.ts.type = timestamp
    cygnusagent.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.GroupingInterceptor$Builder
    cygnusagent.sources.http-source.interceptors.gi.gropuing_rules_conf_file = /usr/cygnus/conf/grouping_rules.conf
cygnusagent.channels.ckan-channel.type = memory
cygnusagent.channels.ckan-channel.capacity = 1000
cygnusagent.channels.ckan-channel.transactionCapacity = 100

# ============================================
# OrionCKANSink configuration
# channel name from where to read notification events
cygnusagent.sinks.ckan-sink.channel = ckan-channel

# sink class, must not be changed
cygnusagent.sinks.ckan-sink.type = com.telefonica.iot.cygnus.sinks.OrionCKANSink

# true if the grouping feature is enabled for this sink, false otherwise
cygnusagent.sinks.ckan-sink.enable_grouping = true

# true if lower case is wanted to forced in all the element names, false otherwise
cygnusagent.sinks.hdfs-sink.enable_lowercase = false

# the CKAN API key to use
cygnusagent.sinks.ckan-sink.api_key = 436fffc8-b397-478a-92fd-bbc5ffaf8269

# the FQDN/IP address for the CKAN API endpoint
cygnusagent.sinks.ckan-sink.ckan_host = ckan-demo.ckan.io

# the port for the CKAN API endpoint
cygnusagent.sinks.ckan-sink.ckan_port = 80

# Orion URL used to compose the resource URL with the convenience operation URL to query it
cygnusagent.sinks.ckan-sink.orion_url = http://localhost:1026

# how the attributes are stored, either per row either per column (row, column)
cygnusagent.sinks.ckan-sink.attr_persistence = column

# enable SSL for secure Http transportation; 'true' or 'false'
cygnusagent.sinks.ckan-sink.ssl = false

# number of notifications to be included within a processing batch
cygnusagent.sinks.ckan-sink.batch_size = 100

# timeout for batch accumulation
cygnusagent.sinks.ckan-sink.batch_timeout = 60

# number of retries upon persistence error
cygnusagent.sinks.ckan-sink.batch_ttl = 10

The file rights seems ok to me:

[root@VM013cen-Prod conf]# ls *.conf -l
-rwxrwxrwx 1 cygnus cygnus 2675 may  3 06:45 agent_test.conf
-rwxrwxrwx 1 cygnus cygnus  258 may  3 05:08 grouping_rules.conf
-rwxr-xr-x 1 cygnus cygnus  135 mar  1 02:50 krb5_login.conf

The test pases ok:

Results :

Tests run: 80, Failures: 0, Errors: 0, Skipped: 0

    [INFO] ------------------------------------------------------------------------
    [INFO] BUILD SUCCESS
    [INFO] ------------------------------------------------------------------------
    [INFO] Total time: 1:05.862s
    [INFO] Finished at: Tue May 03 06:53:44 CDT 2016
    [INFO] Final Memory: 41M/105M
    [INFO] ------------------------------------------------------------------------

The startup command i use: /usr/cygnus/bin/cygnus-flume-ng agent --conf /usr/cygnus/conf/ -f /usr/cygnus/conf/agent_test.conf -n cygnusagent -Dflume.root.logger=INFO,console

0

1 Answer 1

2

Yes, you are right, Grouping Rules are designed for achieving use cases as the one you are describing. So, by defining a simple rule with a regex matching the type of all the entities (assuming such a type is always the same) should do the trick; or by using a regex matching a common part of the entity ID (as you propose).

Nevertheless, your problem seems to be with the Grouping Rules read themselves. You say your configuration contains:

cygnusagent.sources.http-source.interceptors.gi.gropuing_rules_conf_file = /usr/cygnus/conf/grouping_rules.conf

Can you confirm before that line you have something like this?

cygnus-ngsi.sources.http-source.interceptors = ts gi
cygnus-ngsi.sources.http-source.interceptors.ts.type = timestamp
cygnus-ngsi.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.GroupingInterceptor$Builder

In addition, can you check the permissions of /usr/cygnus/conf/grouping_rules.conf? Maybe it cannot be read by the user running Cygnus.

Regarding the rule, the new FIWARE service path should start with /, since all the FIWARE service paths must start with slash (I think this is not properly documented in the last version).

HTH!

EDIT 1:

I think I've found the problem. The configured parameter name is "...gropuing...", while the correct one is "...grouping..." :)

4
  • I edited the question with more info. I also changed the service path adding the /. The problem persist don't know why. Thanks for the answer. May 3, 2016 at 12:27
  • Which version of Cygnus are yoy running? Can you add a pastebin with all the logs of a Cygnus run, please? I'm interested in the traces written from the start of the process.
    – frb
    May 3, 2016 at 12:54
  • OMG! I'm such an idiot. Thanks you so much. Please correct it in the doc so other people don get confused to. Here is the url fiware-cygnus.readthedocs.io/en/latest/quick_start_guide/… were i get the wrong config. May 3, 2016 at 15:14
  • 1
    Not your fault, but mine :) I'll fix the doc.
    – frb
    May 3, 2016 at 20:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.