perl nested regex -

text record:

(1,2,3,4,{fred,don,max,rat,grp},45,67,mat,jhon,{a,b,1,2,sd[{1,2},{4,5}],45,67,p[{34,56,34},{uni,cast,r}],c{q,ew,3,4},1,2,3,cf{2,4,5,8},6},4,fr{24,45,67},5,nol) 

desired output:

(1,2,3,4,{fred:don:max:rat:grp},45,67,mat,jhon,{a:b:1:2:sd[{1:2}:{4:5}]:45:67:p[{34:56:34}:{uni:cast:r}]:c{q:ew:3:4}:1:2:3:cf{2:4:5:8}:6},4,fr{24:45:67},5,nol) 

needs replace : below:

1. everything between {} , []
2. the data has nested structure {{}} , {[],[]}
3. the text record between ().

any perl appreciated.

you may try below perl command uses positive lookahead based regex.

perl -pe 's/,(?=(?:\{[^{}]*\}|[^{}])*})|,(?=(?:\[[^\[\]]*\]|[^\[\]])*\])/:/g' file 

demo