Brown held up his hands and credited the Daggers after two Tresor Kandol goals saw his troops crash to a fifth away defeat of the season.
But despite the result and subsequent effect on the league placings, the boss was right to offer a hand of consolation to his troops, who put on a better performance only to be denied that crucial rub of the green.
Brown said: "I was very proud of the performance of the players today. We got the ball down and tried to play as and when we could.
"It was a very exciting game and end-to-end stuff. I was very disappointed with the first goal we gave away because it was another over-elaborate bit of play in midfield.
"Goals change games and that meant we were always chasing. We had to gamble a bit in the second half and Dagenham are well equipped to hit sides on the break and that led to a great second goal, it has to be said.
"If we'd have won today it would have been three out of four but as it is, it's back to the drawing board and it seems like we have to pick the players up after every game."
True as that might be, Brown has a lot more to be encouraged about this weekend as he starts to get tough on the men who have failed him thus far.
Experienced full back Tarkan Mustafa missed out on the chance to face his former club having been put on the transfer list this week, while decisions to leave out Ahmed Deen and Lewis Hamilton from the 16-man squad were just about exonerated.
The return to the side of Gary Holloway is a massive boost to Brown as he continues to impress, while new loan striker Omari Coleman also looks a useful addition.
The big and powerful Coleman arrived at the Rec on Thursday on a month's loan deal from Lincoln City, having previously enjoyed spells at Watford and Dulwich Hamlet.
Coleman was given the nod to play alongside Tim Sills in place of fellow loanee Chris McPhee, whose spell from Brighton which ends after Saturday's home clash with Kidderminster, looks to be petering out into a disappointing one.
Indeed, Coleman might have rewarded the boss' faith as early as the fourth minute when meeting Holloway's centre from the right, only to fire the ball over the top with an over-hit volley.
Then, on 17 minutes, he came within a whisker of breaking the deadlock again, latching onto a Matt Somner long pass and poking the ball past the advancing keeper Tony Roberts, only, though, to see it ricochet agonisingly off the foot of a post.
The Shots enjoyed the best of the opening 20 minutes, but slowly Dagenham started to work their way back into the game with their ability to play on the counter attack an obvious wrench to the Aldershot's defence.
Injuries meant that Daggers boss John Still only had 15 fit first team players to chose from, four of which being strikers. Indeed, between the likes of Kandol and the supporting trio of Chris Moore, Craig Mackail-Smith and young Paul Benson, the Daggers were a threat throughout.
None of this was more evident than the passage of play which led to their opening goal on 26 minutes.
Just two minutes after Mackail-Smith could only find the legs of the sprawling Nikki Bull when through on goal, the Shots defence were put under pressure again — this time by an uncharacteristic error from Steve Watson. The skipper, caught in two minds, allowed Kandol to dispossess him and after a one-two with Benson, slotted the ball past Bull and into the corner of the net.
The goal came slightly against the run of play as the Shots looked equally threatening — especially from set-pieces.
Impressive defensive duo Greg Heald and Scott Guyett both came close to heading home from Crittenden corners, while only a miracle piece of defending from Anwar Uddin prevented Coleman from stealing a yard and capitalising on a long ball forward.
Darren Barnard almost gave the Shots the perfect half time tonic deep into first half injury time with a free-kick from the edge of the box, which Roberts brilliantly tipped round the post.
With the Shots asking most of the questions, Brown enjoyed a more restrained half-time debate — and it seemed to do the trick.
They started the second half in much the same vein as the first — Sills heading just wide from Crittenden's corner and Guyett firing a volley over the top when Watson's quickly-taken free-kick had pin-pointed him in a good scoring position.
But not too dis-similar to the first half, the Shots were caught on the break and it was Daggers fans left celebrating.
Confusion between Coleman and Andre Boucaud in chasing Crittenden's deep cross led to Ranbir Marwa clearing his lines down the right flank. The ball fell to Mackail-Smith, who with a touch of genius in controlling the ball from behind him and picking out his pass, gave Kandol a hope.
The former Thurrock hit man beat Bull to the chase and finished off the move with true aplomb, slotting the ball into the empty net at the near post.
Shell-shocked Brown introduced McPhee at the expense of the tiring Coleman, but it failed to produce any fresh spark as the Daggers continued to punish their visitors with their counter-attacking ethic.
Scott Griffiths ran a full 60 yards down the left flank, only to fire well over the top, while Bull did well to hold onto Mackail-Smith's 20-yard effort.
But what Dagenham v Aldershot clashes do bring fans are glimpses of, arguably, the two best keepers in the Conference, and in 30 gripping seconds in the very last minute, both Roberts and Bull brought every one of the 1,512 fans to their feet with stunning saves.
Former Welsh international Roberts was the first to show, brilliantly keeping out another Sills header from close range, and from the resulting counter attack, Bull pulled off a fantastic save at his near post after Moore had danced into the Shots penalty area.
Not surprising really when you consider that Bull learnt his trade under the coaching of Roberts whilst at QPR!
This defeat without doubt increases the pressure on Brown, but afterwards the Shots boss off laughed off suggestions of a walk-out from the Rec in typically amiable fashion, and a welcome smile.
He said: "I'm a scrapper and I've been in a lot more difficult situations like this before, believe me. This is relatively an easy job at the moment because we've got quality players getting themselves fit, and when they are I'm sure we will start a rapid climb up the table. If we keep performing like that, we haven't got any problems."
SHOTS: Bull 7; Barnard 6, HEALD 8, Guyett 8, Somner 6, Watson 6 (sub Scott 78mins, 7), Holloway 7 (Reed 88), Boucaud 6, Crittenden 7, Coleman 7 (McPhee 70, 6), Sills 6. Not used: Sulaiman, Weait.
With thanks to the Aldershot News.