Connor had colic really bad till he was 6-7 months old. the only place he would nap was his swing, and he would only sleep half the night in his crib and the second half in his swing.
by all means let him sleep! you can slowly stop it when he gets to the 6 month old and start transitioning naps into his crib will be a lot easier then!
while they are this little, just go with the flow.