Wednesday 9 March 2011

Apache Cassandra & nodetool

Today I was playing around with a cluster of 3.  How can I have new apache cassandra instances join and leave and allow the cluster add/remove to function somewhat autonmous.  Joining is not a problem.  The auto-bootstrap feature works a treat.  Leaving on the other hand.

$CASSANDRA_HOME/bin/nodetool -h 10.0.0.1 decommission

As I understand, when this is run, the node will stream all it's data out to other nodes in the cluster and remove itself.

When I view the ring from another node:


10.0.0.1    Down   Leaving 218.71 KB       21.76%  61078635599166706937511052402724559481

I see this message.  Great ... but after an hour of waiting, I give up and try to force the removal of the token:

nodetool -h 10.0.0.2 removetoken 61078635599166706937511052402724559481
Exception in thread "main" java.lang.UnsupportedOperationException: Node /10.0.0.1 is already being removed.

Ok then... this is interesting:  

nodetool -h 10.0.0.2 removetoken status
RemovalStatus: No token removals in process.

I don't get it.  How do I gracefully remove a node?  Finally, I killed the node on 10.0.0.1 and removed it's data.  Ungraceful.  I then went to the other nodes, still couldn't force it's removal.  Started the node back up on 10.0.0.1 and it's rejoined the cluster ... with data spread evenly around.  Not exactly what I wanted ... oh well....

I'm sure I've missed a concept.  So, now that I have a 3 node cluster working and balanced, I turn off cassandra on 10.0.0.1 and check the ring from another node:

nodetool -h 10.0.0.2 ring

10.0.0.3  Up     Normal  224.21 KB       40.78%  24053088190195663439419935163232881936
10.0.0.1    Down   Normal  213.51 KB       36.78%  86624712919272143003828971968762407027
10.0.0.2    Up     Normal  244.42 KB       22.44%  124804735337540159479107746638263794797

Now, to try and remove that node by removing the token:

nodetool -h 10.0.0.1 removetoken 86624712919272143003828971968762407027

Job done, the node is gone...

nodetool -h 10.0.0.2 ring
10.0.0.3  Up     Normal  224.21 KB       40.78%  24053088190195663439419935163232881936
10.0.0.2    Up     Normal  244.42 KB       59.22%  124804735337540159479107746638263794797

-sd

-- maybe this is my problem:  CASSANDRA-2072

No comments: