Wednesday 9 March 2011

Apache Cassandra & nodetool

Today I was playing around with a cluster of 3.  How can I have new apache cassandra instances join and leave and allow the cluster add/remove to function somewhat autonmous.  Joining is not a problem.  The auto-bootstrap feature works a treat.  Leaving on the other hand.

$CASSANDRA_HOME/bin/nodetool -h decommission

As I understand, when this is run, the node will stream all it's data out to other nodes in the cluster and remove itself.

When I view the ring from another node:    Down   Leaving 218.71 KB       21.76%  61078635599166706937511052402724559481

I see this message.  Great ... but after an hour of waiting, I give up and try to force the removal of the token:

nodetool -h removetoken 61078635599166706937511052402724559481
Exception in thread "main" java.lang.UnsupportedOperationException: Node / is already being removed.

Ok then... this is interesting:  

nodetool -h removetoken status
RemovalStatus: No token removals in process.

I don't get it.  How do I gracefully remove a node?  Finally, I killed the node on and removed it's data.  Ungraceful.  I then went to the other nodes, still couldn't force it's removal.  Started the node back up on and it's rejoined the cluster ... with data spread evenly around.  Not exactly what I wanted ... oh well....

I'm sure I've missed a concept.  So, now that I have a 3 node cluster working and balanced, I turn off cassandra on and check the ring from another node:

nodetool -h ring  Up     Normal  224.21 KB       40.78%  24053088190195663439419935163232881936    Down   Normal  213.51 KB       36.78%  86624712919272143003828971968762407027    Up     Normal  244.42 KB       22.44%  124804735337540159479107746638263794797

Now, to try and remove that node by removing the token:

nodetool -h removetoken 86624712919272143003828971968762407027

Job done, the node is gone...

nodetool -h ring  Up     Normal  224.21 KB       40.78%  24053088190195663439419935163232881936    Up     Normal  244.42 KB       59.22%  124804735337540159479107746638263794797


-- maybe this is my problem:  CASSANDRA-2072

No comments: