No, unless you are shearing your own organically-raised free-range sheep with manual clippers, prepping and spinning your own wool, and are happy to only have clothes in the natural colours that come off your sheep. Or, if you’re bartering your organically-grown vegetables (or whatever) with a neighbour within walking distance who does.
If you want to use plant fibres, even those grown, prepped and spun by you or your neighbour, then you’re pretty much straight-up damaging the environment from the get-go. Cotton is hellaciously bad for the environment, linen may be somewhat less damaging – though your neighbours will be less willing to barter with you after you’ve filled the air with the stench of rotting flax plants.
I hope I don’t have to explain why synthetic fibres are bad for the environment.
I don’t, do I?
You can minimise the environmental impact by buying your yarn from local producers, and looking particularly for yarns that have been dyed with natural, or at least minimally damaging dyes: for example, vinegar can be used to fix dye in animal fibres, though the colours are likely to fade over time. You can also mitigate the damage by reusing yarn from old garments.
However, by making your own clothes, whether knit, crocheted, or woven and sewn, you are helping the environment. For example, you will probably make better-quality clothes, which will last much longer than Primark’s finest. If you do somehow manage to wear out a home-sewn tweed jacket, much of the material can be repurposed for quilts, children’s clothes, and as patches for other clothes – avoiding landfill, and saving you money in the long run. In addition, fabrics like wool don’t require regular washing, so you’re cutting down on the water, electricity and detergent you use: if it gets muddy, just let it dry and then brush it off; if it gets sweaty, hang it in a draught for an hour or so; if it gets soaked, lay it flat near a heat source. A couple of sheep will keep your lawn down with less impact on the environment than a lawnmower.
And so forth. It’s about finding a balance.