Troubleshooting the Activity Engine and Activity Streams

File Handle Limits Exceeded

If your Activity Engine has exceeded its open file handle limits, you may see output like the following:

2013-06-27 04:10:25,894 [pool-7-thread-9] INFO [67073a51-1525-4ca9-80d0-382b2ae58899] com.jivesoftware.eae.service.dao.impl.StreamConfigurationDAOImpl - Added missing stream source connections for user User{userId=43415}
java.io.IOException: Map failed
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
        at com.jivesoftware.eae.stream.kvindex.KeyIndexer.mmap(KeyIndexer.java:238)
        at com.jivesoftware.eae.stream.kvindex.KeyIndexer.index(KeyIndexer.java:216)
        at com.jivesoftware.eae.stream.kvindex.KeyIndexer.get(KeyIndexer.java:165)
        at com.jivesoftware.eae.stream.kvindex.RevolvingKeyIndexer$Door.get(RevolvingKeyIndexer.java:228)
        at com.jivesoftware.eae.stream.kvindex.RevolvingKeyIndexer.get(RevolvingKeyIndexer.java:147)
        at com.jivesoftware.eae.stream.lucene.LuceneFilteredStreamService.handle(LuceneFilteredStreamService.java:254)
        at com.jivesoftware.eae.service.dao.impl.StreamsServiceDAOImpl.dispatch(StreamsServiceDAOImpl.java:115)
        at com.jivesoftware.eae.service.impl.ExclusionManager.trackObject(ExclusionManager.java:218)
        at com.jivesoftware.eae.service.processing.activity.ExclusionHandler.handleActivity(ExclusionHandler.java:102)
        at com.jivesoftware.eae.service.processing.activity.ExclusionHandler.handleActivity(ExclusionHandler.java:20)
        at com.jivesoftware.eae.service.queue.AbstractPipelineProcessor.handlePipeline(AbstractPipelineProcessor.java:160)
        at com.jivesoftware.eae.service.queue.AbstractPipelineProcessor.process(AbstractPipelineProcessor.java:63)
        at com.jivesoftware.eae.service.queue.PhasedQueueConsumerRunnable.run(PhasedQueueConsumerRunnable.java:62)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.OutOfMemoryError: Map failed
        at sun.nio.ch.FileChannelImpl.map0(Native Method)
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:745)
        ... 14 more

In this case, you'll want to increase the soft and hard file handle limits as follows:

Edit /etc/security/limits.conf as follows (note that your file may use the "*" wildcard to specify all users instead of the "jive" user as shown here -- either way, the fix is the same):
```
                jive    soft    nofile      200000
                jive    hard    nofile      200000
```
Verify that the new settings have taken effect:
```
$ su jive -c "ulimit -n"
```
Edit /etc/sysctl.conf to increase the vm.max_map_count. (For most large implementations 500,000 is a good value, but some installations require a higher value depending on usage patterns or overall usage.) It should look like this:
```
vm.max_map_count = 500000
```
Reload:
```
$ sysctl -p
```
Verify that the new settings have taken effect:
```
$ sysctl vm.max_map_count
```

What Matters Stream(s) Not Working Properly

If a stream is not working properly, it may be due to one of the following. Note that on the Admin Console: System > Settings > Activity Engine page, servers shown in yellow are down.

The Jive web application node may not be able to connect to the Activity Engine server. Test the connection by clicking Test Connection. If this test is successful, the Activity Engine server can properly process activity. If this test fails, you'll see a failure message describing the problem. Typically, test failures are caused by the Activity Engine database being down or unavailable. Note that your instance of Jive may not allow you access to the Activity Engine's core server. You'll need to troubleshoot to bring the Activity Engine back online before running another test.
The Activity Engine server may not be able to connect to its database. Verify that the Activity Engine database has been created and you can connect to it.
The Activity Engine server may not be able to connect to the Jive Genius Recommender. Try pinging the Recommender from the Activity Engine.

If you have fixed and verified these connections but continue to experience problems, make sure that all of the configuration settings are correct.

Long Delays Between Post and Stream Publication Times

If there are long delays between post and stream publication times:

Try sending a trace through the Activity Engine by clicking Send Connectivity Trace on the Admin Console: System > Settings > Activity Engine page. This sends a blank activity through the Activity Engine processing pipelines, and then reports the length of time it took to complete the trip. If you send the trace and get back a valid time measurement, the Activity Engine node is up and functioning properly. Typical return times on an idle system would be a few seconds.
Make sure that the web node(s) and the Activity Engine node(s) are configured to use NTP and share very similar, if not identical, times. If they are out of sync, you may see delays.

Log Files Related to the Activity Engine

For further troubleshooting, look at the error messages in the application logs under Admin Console: Management > Logging Management > Log Viewer, or view the application log directly in /usr/local/jive/services/eae-service/var/logs on the Activity Engine node.

eae-service.log -- this is the output of all log statements in the Activity Engine server. Server errors will be logged here.
eae-service.out -- this holds anything written to the standard output stream from the Activity Engine server. Initialization and out-of-memory errors will likely be logged here.
eae-service-gc.log -- analyze this log to determine garbage collection effectiveness and performance.