Here, "silk bamboo" refers to the music that adds to the fun of parties and banquets in the officialdom, and the contrast between "without silk bamboo" and the next sentence "without paperwork" shows that Liu Yuxi (who has been demoted several times) should be somewhat disheartened by all kinds of hypocrisy in the officialdom. From this, we can see that what he doesn't like is the hypocritical entertainment and the dull and tedious life in officialdom, rather than the music (musical instrument) that he really doesn't like. In other words, it is not "silk bamboo" that really confuses the ears, but the flashy hypocrisy hidden behind "silk bamboo".