$CASSANDRA_HOME/bin/nodetool -h 10.0.0.1 decommission
As I understand, when this is run, the node will stream all it's data out to other nodes in the cluster and remove itself.
When I view the ring from another node:
10.0.0.1 Down Leaving 218.71 KB 21.76% 61078635599166706937511052402724559481
I see this message. Great ... but after an hour of waiting, I give up and try to force the removal of the token:
nodetool -h 10.0.0.2 removetoken 61078635599166706937511052402724559481
Exception in thread "main" java.lang.UnsupportedOperationException: Node /10.0.0.1 is already being removed.
Ok then... this is interesting:
nodetool -h 10.0.0.2 removetoken status
RemovalStatus: No token removals in process.
I don't get it. How do I gracefully remove a node? Finally, I killed the node on 10.0.0.1 and removed it's data. Ungraceful. I then went to the other nodes, still couldn't force it's removal. Started the node back up on 10.0.0.1 and it's rejoined the cluster ... with data spread evenly around. Not exactly what I wanted ... oh well....
I'm sure I've missed a concept. So, now that I have a 3 node cluster working and balanced, I turn off cassandra on 10.0.0.1 and check the ring from another node:
nodetool -h 10.0.0.2 ring
10.0.0.3 Up Normal 224.21 KB 40.78% 24053088190195663439419935163232881936
10.0.0.1 Down Normal 213.51 KB 36.78% 86624712919272143003828971968762407027
10.0.0.2 Up Normal 244.42 KB 22.44% 124804735337540159479107746638263794797
Now, to try and remove that node by removing the token:
nodetool -h 10.0.0.1 removetoken 86624712919272143003828971968762407027
Job done, the node is gone...
nodetool -h 10.0.0.2 ring
10.0.0.3 Up Normal 224.21 KB 40.78% 24053088190195663439419935163232881936
10.0.0.2 Up Normal 244.42 KB 59.22% 124804735337540159479107746638263794797
No comments:
Post a Comment